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DETAILED ACTION 

Drawings 

1. The drawings were received on 1/18/2008. These drawings are acceptable. 

Cla im Rejections - 35 USC § 103 

2. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

3. Claims 1-4, 7-10, 16-18, 20-21, 28-30, 32-33 and 40 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Horvitz et al. (Hereinafter "Horvitz": US 
6232974) in view of Itti et al. (Hereinafter "Itti": L. Itti, and C. Koch, "Computational 
Modelling of Visual Attention", Nature Reviews/Neuroscience, Vol. 2, March 2001, pp 1- 
11), and further in view of Slatter (US 7248294). 

Regarding claim 1, Horvitz teaches a method comprising: 
modeling an image with respect to multiple visual attentions to generate a 
respective set of attention objects (AOs) for each attention of the visual attentions (Fig. 
8 and column 7, lines 12-20); and 

analyzing the attention objects and corresponding attributes to optimize a rate of 
information gain as a function of information unit cost in terms of time (column 6, lines 
30-46) associated with multiple image browsing modes (column 14, lines 63-67), 
wherein the corresponding attributes for each attention object of the AOs comprise a 
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minimal perceptible time (MPT) for display of subject matter associated with the 
attention object (column 17, lines 65-67 and column 18, lines 1-2). 

Horvitz does not teach, responsive to analyzing the attention objects, generating 
a browsing path to select ones of the attention objects. 

However, Itti discloses a method of, responsive to analyzing the attention 
objects, generating a browsing path (i.e. scanpath) to select ones of the attention 
objects (page 3, left column, lines 2-8). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have combined Itti's method with Horvitz's teaching, 
the motivation to combine being to enhance understanding of a visual scene (Itti: page 
2, left column, lines 6-9). 

Horvitz combined with Itti does not teach the limitation of causing display of 
subject matter associated with each of the selected ones of the attention objects for at 
least a respective minimal perceptible time by panning the image according to the 
generated browsing path. 

However, Slatter discloses an imaging system wherein subject matter associated 
with each of the selected ones of the attention objects (column 2, lines 7-14, i.e. faces) 
for at least a respective minimal perceptible time (i.e. time required for faces to be 
clearly seen) by panning the image (column 3, lines 8-13). 

It would be obvious to one of ordinary skill in the art at the time of the invention 
to combine Slatter's panning method with the Horvitz/Itti browsing system in order to 
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clearly see picture details when a picture is too large for the display device (Slatter: 
(column 1, lines 13-21). 

Regarding claim 2, Horvitz in combination with Itti and Slatter teaches a method 
as recited in claim 1, wherein the multiple visual attentions are based on saliency 
(Horvitz: column 24, line 22), face (Itti: page 8, right column, line 9), and text (Itti: 
page 7, right column, line 25-26, i.e. recognition of words) attention models. 

Regarding claim 3, Horvitz in combination with Itti and Slatter teaches a method 
as recited in claim 1, wherein the image browsing modes comprise a perusing mode 
(Horvitz: column 14, lines 66-67, i.e. selectively attending) and a skimming (Horvitz: 
column 14, lines 63-66, i.e. scalar model) mode. 

Regarding claim 4, Horvitz in combination with Itti and Slatter teaches a method 
as recited in claim 1, wherein the select ones have relatively greater information fidelity 
as compared to different ones of the attention objects (Horvitz: Fig. 11). 

Regarding claim 7, Horvitz in combination with Itti and Slatter teaches a method 
as recited in claim 1, wherein modeling further comprises: creating a visual attention 
model for the image, the visual attention model being generated according to 

{AO} = {(ROI /r AV h MPS,-, MPT)}, 1 < /< N (Horvitz: Fig. 1, 14a, 16a, 18a, to 
14c, 16c, 18c); and, wherein AO,, represents an P AO within the image (Horvitz: 
column 9, lines 58-60, i.e. component), ROI, represents a region-of-interest of AO-, 
(Horvitz: column 7, line 17, i.e. geometric level of detail), mis represents an attention 
value of AO-, (Horvitz: column 6, line 31, i.e. expected perceptual cost), MPS, represents 
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a minimal perceptible size of (Horvitz: column 7, line 17, i.e. spatial resolution), 
MPT/ represents a minimal perceptual time for display of subject matter associated with 
the AO-, (Horvitz: column 17, line 66), and, N represents a total number of AOs modeled 
from the image. 

Regarding claim 8, Horvitz in combination with Itti teaches a method as recited 
in claim 1, wherein the browsing path comprises a number of successive path segments 
described as follows: P= = {{SP h EP h SR/ER/, T])}, 1 < /< N, and wherein P, 
represents an /"path segment, SP, represents a starting point of P„ ^corresponds to 
an ending point of P h SR, is a starting resolution of P„ ER, is an ending resolution of P-, 
and T-, is a time cost for scrolling from SP-, to Ep,- (Horvitz: column 7, lines 15-19, i.e. 
spatial resolution. The examiner notes that a starting point, end point and a time cost 
for scrolling are inherent properties of a browsing path segment). 

Regarding claim 9, Horvitz in combination with Itti and Slatter teach a method 
as recited in claim 1, wherein generating the browsing path further comprises creating 
the browsing path as a function of a fixation state (Horvitz: column 14, line 67, i.e. 
attending to a specific component) or a shifting state (Horvitz: column 14, lines 64-66, 
i.e. scalar model). 

Regarding claim 10, Horvitz in combination with Itti and Slatter teach a method 
as recited in claim 1, wherein generating the browsing path further comprises 
calculating an information fidelity for each AO, the information fidelity being a function 
of an attention value (AV) and the minimal perceptible time (MPT) for display of subject 
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matter associated with the AO (Horvitz: column 7, lines 38-43, i.e. cost-benefit analysis 
for temporal resolution). 

Regarding claims 16, Horvitz in combination with Itti and Slatter discloses a 
computer-readable medium (Horvitz: Fig. 13, Hard Drive 327) comprising computer- 
program instructions executable by a processor for executing the methods as recited in 
claims 1 and 9 above. 

Regarding claim 17, Horvitz in combination with Itti and Slatter discloses a 
computer-readable medium as recited in claim 16, wherein the multiple visual 
attentions are based on saliency, face, and text attention models (as recited in claim 2 
above). 

Regarding claim 18, Horvitz in combination with Itti and Slatter discloses a 
computer-readable medium as recited in claim 16, wherein the select ones have 
relatively greater information fidelity as compared to different ones of the attention 
objects (as recited in claim 4 above). 

Regarding claim 20, Horvitz in combination with Itti and Slatter discloses a 
computer-readable medium as recited in claim 16, wherein the computer-program 
instructions further comprise instructions for generating the browsing path as a number 
of successive path segments as recited in claim 8 above. 

Regarding claim 21, Horvitz in combination with Itti and Slatter discloses a 
computer-readable medium as recited in claim 16, wherein the computer-program for 
generating the browsing path further comprise instructions for calculating an 



Application/Control Number: 10/688,471 Page 7 

Art Unit: 2624 

information fidelity for each AO, the information fidelity being a function of an attention 
value (AV) and the minimal perceptible time (MPT) for display of subject matter 
associated with the AO (as recited in claim 10 above). 

Regarding claim 28, Horvitz in combination with Itti and Slatter discloses a 
computing device comprising a processor coupled to a memory, the memory comprising 
computer-program instructions executable by the processor (Horvitz: Fig. 3 Personal 
computer 320) for executing the methods as recited in claims 1 and 9 above. 

Regarding claim 29, Horvitz in combination with Itti and Slatter discloses a 
computing device as recited in claim 28, wherein the multiple visual attentions are 
based on saliency, face, and text attention models (as recited in claim 2 above). 

Regarding claim 30, Horvitz in combination with Itti and Slatter discloses a 
computing device as recited in claim 28, wherein the select ones have relatively greater 
information fidelity as compared to different ones of the attention objects (as recited in 
claim 4 above). 

Regarding claim 32, Horvitz in combination with Itti and Slatter discloses a 
computing device as recited in claim 28, wherein the computer-program instructions 
further comprise instructions for generating the browsing path as a number of 
successive path segments as recited in claim 8 above. 

Regarding claim 33, Horvitz in combination with Itti and Slatter discloses a 
computing device as recited in claim 28, wherein the computer-program for generating 
the browsing path further comprise instructions for calculating an information fidelity for 
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each AO, the information fidelity being a function of an attention value (AV) and the 
minimal perceptible time (MPT) for display of subject matter associated with the AO (as 
recited in claim 10 above). 

Regarding claim 40, Horvitz, in combination with Itti and Slatter discloses a 
computing device (Horvitz: Fig. 3 Personal computer 320) comprising the recited 
means. 

4. Claims 6, 14, 27 and 39 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Horvitz combined with Itti and Slatter as applied to claim 1 above, 
and further in view of Boguraev et al. (Hereinafter "Boguraev": US6353824). 

Regarding claim 6, Horvitz in combination with Itti and Slatter teaches a method 
as recited in claim 1, wherein the corresponding attributes for each attention object of 
the attention objects comprise a minimal perceptible time (MPT) for display of subject 
matter associated with the attention object (Horvitz: column 17, lines 65-67 and column 
18, lines 1-2). 

Horvitz in combination with Itti and Slatter does not teach the further limitation 
of calculating the MPT as a function of: a number of words in the subject matter; 
whether the subject matter is for presentation to a viewer in a perusing image browsing 
mode or a skimming image browsing mode; user preferences; and/or, display context. 
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However, Boguraev discloses a method wherein calculating the MPT is a function 
of a number of words in the subject matter (column 20, lines 6-8, i.e. amount of 
information). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have combined Boguraev's method with the combined 
teaching of Horvitz, Itti and Slatter, the motivation to combine being to improve 
browsing efficiency (Boguraev: column 2, lines 30-42). 

Regarding claims 14, 27 and 39, Horvitz in combination with Itti and Slatter 
teaches a method as recited in claim 1, discloses a computer-readable medium as 
recited in claim 16 and a computing device as recited in claim 28. 

Horvitz in combination with Itti and Slatter does not teach the limitations further 
comprising: detecting user intervention during automatic playback of the browsing 
path; responsive to detecting the user intervention: recording all AOs S.sub.r of the AOs 
that have not been browsed; identifying all AOs S.sub.m of the AOs browsed during the 
user intervention; regenerating the browsing path based on S.sub.r-S.sub.m; and 
responsive to regenerating the browsing path and determining that there is at least a 
lull in user intervention, automatically navigating the browsing path. 

However Boguraev discloses a method comprising: detecting user intervention 
during automatic playback of the browsing path (column 21 lines 3-7, i.e. zoom 
mechanism); responsive to detecting the user intervention: recording all AOs S.sub.r of 
the AOs that have not been browsed; identifying all AOs S.sub.m of the AOs browsed 
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during the user intervention; regenerating the browsing path based on S.sub.r-S.sub.m; 
and responsive to regenerating the browsing path and determining that there is at least 
a lull in user intervention (column 21, lines 41-44, i.e. Screensaver mode), automatically 
navigating the browsing path (column 21, lines 34-37, i.e. default mode). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have combined Boguraev's method with the combined 
teaching of Horvitz, Itti and Slatter, the motivation to combine being to improve 
browsing efficiency (Boguraev: column 2, lines 30-42). 

5. Claims 11, 22 and 34 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Horvitz combined with Itti and Slatter as applied to claim 1 above, in view of Lee 
et al. (Hereinafter "Lee": Keansub Lee, Hyun Sung Chang, Seong Soo Chun, Hyungseok 
Choi and Sanghoon Sull, "Perception-Based Image Transcoding for Universal Multimedia 
Access", School of Electrical Engineering Korea University, Seoul, Korea, 2001 IEEE, pp. 
475-478) and further in view of Boguraev. 

Regarding claim 11, Horvitz in combination with Itti and Slatter teaches a 
method as recited in claim 1. 

Horvitz in combination with Itti and Slatter does not teach the limitations wherein 
each AO is a respective information block and wherein the image (I) is a modeled as a 
set of M*N evenly distributed information blocks Ig as follows: 

I - \iq } « f 4% , r% )), I < t < M »! < J ^ N t r gi € (U) , 
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wherein (i, j) corresponds to a location at which the information block Iy is 
modeled according to a visual attention, AVy is a visual attention value of I,, rg is the 
spatial scale of Iy, representing the minimal spatial resolution to keep Iy perceptible; 
and, wherein generating the browsing path further comprises calculating an information 
fidelity (f RS vp) for each AO, the information fidelity being calculated as follows for 
respective ones of the information blocks Iy: 



and wherein I RS vp(t) is a subset of the information blocks and varies with time 

and 



which varies with space. 

However, Lee discloses an equivalent method, excluding the time dependency 
(page 476, section 2.1, paragraph 2, section 2.2, and page 477, equations 2-4, i.e. 
visual attention value (AV) is importance value (s), and information value (f) is content 
value (V)). 

Therefore it would have been obvious to combine Lee's teaching with the 
combined teaching of Horvitz, Itti and Slatter, the motivation to combine being to 
provide access to multimedia content from a wide variety of devices (Lee: page 475, 
first paragraph) 



T 
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Lee does not disclose a time dependency for visual attention and spatial scale. 

However, Boguraev discloses a method wherein visual attention and spatial scale 
are time dependent (column 20, lines 6-8, i.e. display time depends on information 
content, and column 21, lines 3-5, i.e. zoom) 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have combined Boguraev's method with the combined 
teaching of Horvitz, Itti, Slatter and Lee, the motivation to combine being to improve 
browsing efficiency (Boguraev: column 2, lines 30-42). 

Regarding claim 22, Horvitz in combination with Itti, Slatter, Lee and Boguraev 
disclose a computer-readable medium as recited in claim 16, wherein each AO is a 
respective information block, and wherein the computer-program instructions further 
comprise instructions for implementing the method of claim 11 above. 

Regarding claim 34, Horvitz in combination with Itti, Slatter, Lee and Boguraev 
disclose a computing device as recited in claim 28, wherein each AO is a respective 
information block, and wherein the computer-program instructions further comprise 
instructions for implementing the method of claim 11 above. 

Response to Arguments 
Summary of Applicants remarks: The limitation of panning the image 
according to browsing path is not found in the prior art of record. Pirolli does not teach 
information fidelity. 
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Examiner's Response: Applicant's arguments with respect to claims 1-4, 6-11, 
14, 16-18, 20-30 and 32-40 have been considered but are moot in view of the new 
ground(s) of rejection. 

Applicant's arguments, see pages 30-34, filed 1/18/2008, with respect to claim 
12 have been fully considered and are persuasive. The rejection of claim 12 has been 
withdrawn. Applicant's arguments above also apply to claim 13 and new claim 41. The 
rejection of claim 13 has been withdrawn. Claim 41 is allowed. 

Allowable Subject Matter 

6. Claims 12, 13, 23-26 and 35-38 are objected to as being dependent upon a 
rejected base claim, but would be allowable if rewritten in independent form including 
all of the limitations of the base claim and any intervening claims. The recited equations 
are neither anticipated nor made obvious by the prior art of record. 

Conclusion 

7. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 

§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
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TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire 
later than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to CLIFTON G. DALEY whose telephone number is 571- 
270-3144. The examiner can normally be reached on Monday - Friday 7:30am - 
4:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Samir Ahmed can be reached on 571-272-7413. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. Status 
information for unpublished applications is available through Private PAIR only. For 
more information about the PAIR system, see http://pair-direct.uspto.gov. Should you 
have questions on access to the Private PAIR system, contact the Electronic Business 
Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO 
Customer Service Representative or access to the automated information system, call 
800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Samir Ahmed 
Examiner 
Art Unit 2624 

CGD 

4/23/2008 



/Samir A. Ahmed/ 

Supervisory Patent Examiner, Art Unit 2624 



